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ABSTRACT 

Pour problems underlying the delivery of technical 
assistance in Elementary Secondary Education Act (ESEA) Title I 
evaluation are discussed: the dual missions of facilitating the 
implementation of the Title I Evaluation and Reporting system and of 
facilitating the local use of evaluation information frequently 
require tradeoffs between evaluation activities; (2) there is a 
relative lack of an experiental base on which technical assistance 
strategies can be built: f3) research on evaluation in its natural 
settina is recent and sketchy: and (U) conceptual analyses of 
technical assistance as a part of planned educational change are not 
available. This paper reviews literature on providing technical 
assistance in evaluation, evaluating educational programs within the 
educational system, and analyzing educational change in terms of 
implementation processes. Models for clarifying evaluation goals and 
developino state and local evaluation plans are described. A model 
for analyzing state and local education agencies* planned evaluation 
system improvements, in terms of stages of implementation or levels 
of use, is discussed as a means for improving the relevance md 
efficiencv of Title I technical assistance. (Author/GDC) 
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Probl^ Statement 

We address a multif aceted problem in this symposium. The following 
conditions are broken out as its key el^ents. 

1. The Technical Assistance Center (TAG) mission involves multiple 
goals awonq which tradeoffs may be necessary . The primary goals of 
Title I Technical Assistance Centers are to facilitate implementation of 
the United States Office of Education's (OSOE) Title I Evaluation and 
ReporUng System (TIERS) and to facilitate local use of Title I 
evaluation. These goals are conceptually distinct. The Title I 
Evaluation and Reporting System has been described by Wisler and Anderson 
(1979) as one which "should satisfy evaluation needs at the local, state 
and federal levels but without necessarily being the sole form of 
evaluation at any of these levels" (p. 49). By implication, 
concentrating technical assistance solely on implementation of the 
Title I Evaluation and Reporting System is not necessarily expected to 
facilitate local use of Title I evaluation. Additional evaluation 
activities may be necessary. 

Two factors compound the condition. First, the technical assistance 
under consideration is in evaluation. Evaluation serves a variety of 
purposes including providing an accountability function. As a 
consequence, providing technical assistance has the potential for role 
conflict. Second, evaluation use is an emerging concept requiring 
further clarification as it becomes a meaningful goal. 
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2. Providing technical assistance in evaluation on a large scale is 
a relatively new enterprise ^ Examples of projects providing evaluation 
technical assistance include the Evaluation Training Consortium project 
sponsored by the Bureau of Education for the Handicapped (Brinkerhof t\ 
1979) f the California Improvement Project initiated In 1974 and funded by 
OSOE {Finley^ 1979), and the Technical Assistance Development System 
providing evaluation assistance to 63 demonstration projects for 
preschool handicapped children {Suarez, 1979). The Title I Evaluation 
TAG network, initiated during 1976, is the largest of these evaluation 
technical assistance programs. 

Each of these projects began after 1974. Consequently, the 
experiential basis for planning and delivering technical assistance in 
evaluation is both recent and limited. 

3. The practice of evaluation has only: recently received attention 
as an area of empirical inquiry . The research of Alkin, Daillak and 
White (1979), David (1978), hycm (1979) and Patton (1978) has focused on 
factors defining and affecting the use of evaluation information in its 
natural organizational setting. Research being conducted by the Huron 
Institute is in early stages but is expected to be used in forming policy 
on future technical assistance strategies (Kennedy, personal 
communication). These studies are very recent and largely exploratory in 
nature, providing only limited information for shaping the delivery of 
technical assistance. 

4. At present there is no commonly accepted conception of 
evaluation technical assistance in relation to educational knowledge 
production ani utilization . Technical assistance may be characterized as 
operating within a change strategy. By implication, a technical 



assistance program may be asked to provide a variety of support services 
that depend upon the particular change strategy being followed, the 
organizational arrangements or other variables. A review of literature 
on technical assistance in relation to educational change strategies 
failed to produce conceptual treatments. Thus, there is a weak 
theoretical base as well as a weak experiential basis to draw on in 
providing quality tecanicaLl assistance. 

At a rainiroun, these four conditions vAiich interact underlie the 
problem we address. The relative lack of research on evaluation as 
practiced in the natural organizational setting contributes to 
uncertainty in making tradeoffs among sometimes competing evaluation 
goals or choosing alternative roles. Research on evaluation use can be 
viewed as a special instance of this larger domain of research on 
knowledge production and utilization. A theory of evaluation use, if 
attainable, would occupy a niche within the larger domain. Hypotheses 
derived from research on knowledge production and utilization might be 
tested within research on evaluation use. Hypotheses suggested frcm 
research on evaluation use might be compared with reigning hypotheses in 
the knowledge production and utilization dcmain. 

These conditions imply different problems to different people. Some 
may emphasize the need for research and better theory? others may seek 
solutions in policy; still others may see these conditions as something 
each technical assistance program must deal with on its own terms. 

The view of the problan state expressed in the paper is that of a 
provider of evaluation technical assistance. The emphasis is placed on 
guides to action, action hypotheses, corresponding procedures and 
self-evaluation techniques and "mind holds" as described by Chin and 
Downey (1973) . 
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In the next section we provide an overview of three general areas of 
inquiry that may provide insights into improving the delivery of 
technical assistance. 

Search for "Mind Holds" 

Guidance Received from Recent Experience in Evaluation Technical 
Assistance 

External views > During the three and one-half year period of TAC 
performance, one formal external evaluation of the TAC system was 
conducted by Hope Associates (1979). * Under the sponsorship of the 
Assistant Secretary for Planning and Evaluation (ASPE) , Department of 
Health, Education and Welfare, the Panel's full charge was to evaluate 
OSOE's implementation of its federal mandate, under Section 151c of ESEA 
Title I as amended, to provide technical and other assistance for Title I 
evaluation. Because of the scope of the TAC system ls compared to other 
implementation activities the Panel concentrated heavily on the TACs, 
intending to identify ways to improve the TAC system. Thus, the Panel's 
investigation was based upon the experience of TACs within the context of 
OSOE's implementation of its legislative mandate. 

To provide a conceptual basis for recommendations, the Panel drew 
upon the general area of knowledge production and utilization* Appendix 
G of the Panel's report sketches out elements of a guiding theory in 
which technical assistance is viewed as "a change system performing a 
communication function directed toward new practice and not solely new 
knowledge* (p. 95). As a consequence of this view, the Panel's 
recommendations focus upon technical assistance strategies and tactics 



rather than on the particular content of the Title I Evaluation and 
Reporting System or evaluation assistance in general. 

The Panel's recoinmendations for improving technical assistance 
through modifying what was then the TAC scope of work include: 

1. " That the taCs be aUowed to provide technical assistance 
services, as negotiated with each state, that assist in the 
development of increased capacity for Ti tle I evaluation. 
Permissible areas of service should place more emu^asis on 
technical assistance directed toward identifying strengths and 
weaknesses o£ individual programs for the purpose of improving 
them" (p. 56). 

2. "That each TACjplace greater emphasis on staff develoffaent 
relevant ':o the provision of technical assistance for Title I 
evaluation* (p. 57). 

3. "That research relevant to Title I evaluation problgas and 
Issues be recognized as a permissible activity for TACs to 
undertake and that a certain proportion of TAC funding be 
earm a rked for such research activiaes. Title I ev aluation 
related activities that require a more specia lized knowledge and 
expertise {e.g., data analys.^-^ . quality control o^ d ata) should 
also be permitted" (p. 57), 

A fourth Panel recommendation made to the OSOE provided added 
emphasis to the first recoromt..dation for formulating the new TAC scope of 
w«rk. The Panel recommended: 

4. "That the Office of Education begin to investig ate, during the 
period of the next contracts for Technical Ass istance Centers, 
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the possibility.' o£ a future systero that has flexibility to 
accocapodatet the diversity of state and local capabilities and 
needs, and also the enlarged objectives of 1-tle I evaluation 
technical assistance* particularly Including the uses of 
evaluation fo r local j)roqram improveroent and the strengthening 
of Ic - nl evaluation capacity* (p. 60) . 
k reading of the Panel's full report is necessary to understand the 
rationale underlying its recommendations which were later acted on in 
USOE's Request for Proposal that initiated the current period of TAG 
funding. An issue surfacing throughout the report is the extent to which 
implementation of the Title I Evaluation and Reporting System will supply 
information useful for local program improvement. 

Based on a series of case studies on local use of Title I evalution, 
David (1978) had previously reconimended that the TAG mission place 
greater emphasis cm local use of evaluation, and, to this end, that 
strategies be designed to encourage local education agencies (LBAs) to 
identify evaluation questions of interest to th^. Still earlier, 
McLaughlin (1975), describing Title I evaluation from 1965 to 1972, 
suggested that the issue of evaluation use at the local level was not 
adequately addressed and was at the root of many of the early evaluation 
failures in Title I. Obviously, the issue is not without historical 
precedent. 

The panel reported that some of those state education agency (SEA) , 
TAG and tJSOE staff interviewed believed that process or formative 
evaluation would be more useful at the local level than models 
implementation. This issue, along with alleged technical weaknesses of 
the evaluation models, contributed co the first and fourth Panel 
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reoonmendacions listed above. A concurrent ASPE-sponsored survey of LEAs 
IdentlCied the area of highest need to be assistance in evaluation for 
program improvement (National Center for Educational Statistics, October 
18, 1979). 

The effect of these two recommendations on the TAC mission is 
noteworthy. In those statas and districts perceiving the Title I 
Evaluation and Reporting System as inadequate for informing program 
improvement, the need is heightened for making tradeoffs between 
facilitating Title I Evaluation and Reporting System implementation and 
facilitating the use of evaluation for local program improvement. The 
Panel's recommendation on this issue has contributed to the severity of 
the problem of multiple goals raised at the beginning of this paper. 
However, rebalancing the two major TAC goals was probably necessary and 
inevitable, given historical circum?tances and current perceptions. 

Wisely, the Panel stressed that its recommendations were permissive 
in nature. The increased emphasis placed on what had been a secondary 
goal was offset by recommendations for staff development and research 
related to fhe provision of technical assistance in Title I evaluation. 
Clearly, the recommendations encouraged TACs to find ways to adapt to 
changing circumstances. 

In addition to the Hope Associates Report (1979) , the ASPE-sponsored 
survey (October 18, 1979) and David's (1978) study, there are about 200 
other papers dealing with recent USOE- sponsored efforts, including the 
TAC system, to improve Title I evaluatitMi. The vast majority of these 
papers deal with technical issuf and informed "points of view" about the 
impact that implementing the Title I Evaluation and Reporting System 
(TIERS) has had upon various agencies. Approximately 15 papers deal with 
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issues related to the delivery of technical assistance^ Since 
perspectives on these issues vary among sponsors, providers and receivers 
of technical assistance^ the review of the papers is organized according 
to the role of the author • 

Sponsor views ,! A number of papers by federal officials contain 
discussions of the broad intent of USOE-sponsored efforts to improve 
Title I evaluation (c.f • Anderson, Fishbein, & Stonehill, 1979; Barnes & 
Ginsburg, 1979; and Wisler & Anderson, 1979)* 

On the theme of TAG goals, Anderson, Pishbein and Stonehill (1979) 
stated: 

Since September, 1976, USOE personnel have placed first priority 
on assisting States to implement the reading, mathematics and 
language arts evaluation models* Without a secure foundation on 
which to base more elaborate and ambitious evaluation strategies, it 
would have proven extremely difficult to organize and disseminate 
evaluation activities aimed at local program improvement* However, 
it should be noLed that although USOE has placed priority on the 
implementation of the TIERS, assistance in the areas of Title I 
evalution which are not addressed by these models has been provided 
to some SEAs and LEAs, and the availability of sue* services has been 
made generally known* 

The models for assessing the achievement impact of Title I 
programs have been published in the Federal Register (February 7, 
1979) , as draft regulations^. The hope is that through application 
of these models at the SEA and LEA levels r USOE will be provided with 
data that can be used to develop a major component of its report to 
Congress on the status of the Title I program. 

Now that all SEAs are preparing to irapl^ent the proposed 
evaluation models, and many have already done so on a statewide basis 
for several years, USCffi can devote more effort to helping States make 
better use of the data that is being produced through the 
implementation of the TIERS and to assist any SEAs or LEAs to develop 
evaluation strategies that will address questions of program 
importance and interest that are not adequately answered by 
implementing the TIERS (p. 6-7). 



^The final evaluation regulations were published in the Federal Register 
(October 12, 1979). 



Anderson et al. {1919) go on to describe the USOE-sponsored efforts 
to increase local utility of evaluation. Although TIERS was not assumed 
to be the sole means of meeting local evaluation needs, it was, 
nonetheless, the priority activity in the early years of implementing 
technical assistance. 

Wisler and Anderson (1979) describe the rationale underlying the 
development of TIERS. Their first premise is: 

"The system should satisfy evaluation needs at the local, state and 
federal levels, but without necessarily being the sole form of 
evaluation at any of those levels" (p. 49). 

This view considers TIERS as a common core, an element in any Title I 
evaluation system defined at an LEA or SEA level, but not a comprehensive 
system at any level. 

Provider views . As providers of Title I evaluation technical 
assistance, TAG staff have accumulated experier.^e and data on the Title I 
evaluation improvement effort {c.f. Bessey, 1979 j Hansen and Oxford, 
1979? Oxford, 1979? Temp, 1979j and Troy, 1979). These papers cite 
improved evaluation practices in states receiving technical assistance. 
Examples include improvements in test selection procedures, test 
interpretation ani scoring, forms and reporting procedures and testing at 
times appropriate for valid normative interpretation (the norm-referenced 
evaluation model). Activities going beyond the routine application of 
TIERS include USOE-sponsored state refinement contracts dealing with 
quality control procedures, providing feedback from SEAs to LEAs and 
exploring procedures for Title I reporting. These are also cited as 
evidence of "impact." 



The evidence of impact comes from a number of sources including 
personal observations, surveys and interviews of SEA and LEA staff, and 
document reviews. Two problems are encountered in interpreting this 
information. First, there is little direct evidence about the actual use 
of evaluation results* Much of what has been reported to date reflects 
plans for using data and approaches to technical assistance hypothesized 
to encourage and facilitate use. Hansen and Oxford (1979) report that 
LEAs responding to a mail survey expressed greater interest in the use of 
data for decision making than did SEA respondents* At the same time, SEA 
and LEA attitudes toward evalution were reported as improving. The 
second and related problem is the difficulty of determining whether 
progress is actually being made. Given TIERS requirements, what changes 
might have been expected and when? Three papers prepared by TAG staff 
begin to identify conceptual views within which one might attempt to 
interpret and evaluate improvement across time* Troy (1979) suggests 
that encouraging the use of evaluation for program improvement requires 
careful planning of the evaluation so that local program evaluation 
activities produce information satisfying the requirements of TIERS as 
well as answering locally defined evaluation questions. 

Bessey (1979) proposed a three-stage process of using evaluation for 
program improvement. The first stage calls for implementing evaluation 
practices allowing one to attribute gains to treatments with confidence. 
Producing internally valid results is the priority objective. The second 
stage of the process involves providing feedback on the project's 
outcomes to the LEA. In the third stage, valid data are used to improve 
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the program or to identify exercplary instructional practices for 
dissemination* 

The major difference between the views of Bessey (1979) and of Troy 
(1979) is that Bessey's three-stage process assumes measurement of 
project achievement qains to be the priority evaluation question-- one 
which presumably has direct and sufficient implications for program 
improvement. Troy (1979), on the other hand, sees a necessity to give 
equal priority during initial planning both to measuring project gains 
and to specifying the local evaluation questions. The dual missions of 
facilitating local use and of TIERS implementation surface again, this 
time in terms of alternative strategies for implementing technical 
assistance. 

What is significant is not so much that Troy (1979) and Bessey (1979) 
have proposed incompatible processes but that both have described 
technical assistance strategies including workshop materials, quality 
control procedures, etc., that correspond to their respective points of 
view. Furthermore, each claims that her strategy meets with client 
acceptance. 

Brinkerhof f (1979) discussed lessons learned through the operation of 
the Evaluation Training Consortium (ETC) project. The ETC project was a 
national project sponsored by the Bureau of Education for the Handicapped 
to upgrade the program evaluation capabilities and performance of special 
education personnel preparation programs. The evaluation training 
provided was based on Provus' Discrepancy Evaluation Model. 
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According to Brinkerhoff technical assistance should be: 
!• Part of a self-help strategy 

2. Integrally linked to treaning 

3. Designed to maintain client investment in all work completed 
4» Model-based 

5» A combined responsive and proactive \ystem 
6. Product related 

Each point has implications for improving Title I evaxuation 
technical assistance. Conceptualizing technical assistance as part of a 
self-help strategy emphasizes the importance of defining roles and 
responsibilities of providers and receivers in such a way that it is 
possible to determine where technical assistance will begin and end. 
Transfer of knowledge and capabilities is expected to result fron the 
training aspect of technical assistance. Maintaining the client's 
investmen^ in all work completed is closely related to the first point. 
Client dependence on the providers of technical assistance must be 
minimized. 

Providing models-based assistance ensures that services are 
comprehensive, internally consistent and goal-oriented. The structure 
ensuring the coherence of the technical assistance does not need to limit 
its re^sponsivenesSf however. Services are initiated by client requests 
and may be planned by the providers of technical assistance to take into 
account the receiver's status vis-a-vis evaluation planning , 
implementation and use. One tangible indication of receiver status is 
the focus of current client activity on evaluation. For example, an LEA 
request for assistance in using TIERS data as input to the programmatic 
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needs assessment demonstrates achievement In evaluation planning Hnd 
iiaplenentation, and a readiness to focus on use. Locating secvicra 
requests within the larger evaluation effort of a particular client 
provides contextual clues to the provider that should make the service 
delivered more tailored and, therefore, more effective. 

On a broader note, an examination of issues in assisting SEAn and 
LEAS in implementing the Title I Evaluation and Reporting System prompted 
Oxford (1979) to conclude that "ultimately the adequacy of the process 
and impact of the Title I Technical Assistance Centers should be triewed 
in light of a change model* (p. 34) . The model suggested by Oxfoird was 
the Research-Develop-D if fuse-Adopt Model proposed by Cuba and Clark 
(undated). In a subsequent section this model and an alternative view 
will be described in more detail. 

Receiver views . Kearns' (1979) survey of state Title I coordinators 
addressed a number of questions regarding the impact (or Implementi tion) 
of the Title. I Evaluation and Reporting System on the SEAs. Changes in 
staff assignments, adding staff, providing inservice training, modifying 
technical assistance activities with local districts, rebudgeting and 
increased effort spent on debating or defending issues unfamiliar to many 
involved in program administration were some of the changes noted. 

Kearns observes that the key value of Title I evaluation lies in its 
ability to provide the local district with information on the results of 
program effectiveness. The ASPE-aponsored survey of LEA needs for 
technical assistance supports this observation (NCES, October 18, 1979). 
The benefits that may derive for states, USOE or Congress are viewed as 
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secondary. In this light, overemphasis on aggregation, which reflects a 
unitary system with a ooramon goal, fails to recognize the diversity of 
goals for evaluation among different users and could result in failure to 
provide locally meaningful evaluation. Attempts to use the TIERS as the 
sole means to meet the needs of all three levels of users may be the 
system's undoing. A factor contributing to resistance to the TIERS is 
the continuing and unsettling debate over technical issues involving tht 
TIERS. Kearns called for rapid resolution of major technical issues and 
for corresponding system modifications. Ironically, the survey 
respondents perceived the quality of TIERS data to be relatively high 
compared to the extent to which TIERS data are provided for improved 
program decision making. To remedy the problem, Kearns identified a need 
Cor methods of process evaluation and longitudinal evaluation appropriate 
to local district Title I programs. 

The previous paragraphs are representative of the points of view 
expressed by external evaluators, sponsors, providers and receivers of 
technical assistance about the relative priority of facilitating TIERS 
implementation and facilitating the use of evaluation for local program 
improveraent. Ad hoc resolution of this issue is a routine aspect of 
providing technical assistance. 

To conclude that the two goals are nonover lapping would be an 
oversimplification. That is, knowing whether a project produced 
achievement gains is of some consequence to those interested in local 
program improvement. On the other hand, it should be clear that the two 
goals are different. Facilitating evaluation for local program 
improvement should be based on questions and issues of relevance to local 
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decision makers. Practical resolution of the issue must be achieved as 
technical assistance is planned and delivered within each receiver 
agency. Since providers of Title I evaluation technical assistance are 
actively involved in a change process, it seems appropriate to turn to 
research on evaluation use and planned educational change as two sources 
of guidance. 
Research on Evaluation 

Providing technical assistance services responsive to client needs 
and circumstances {i.e., client-centered) depends, in part, on knowledge 
of the organizational arrangements and activities associated with the 
clients involved in evaluation. Evaluation activities are carried out by 
people occupying positions within agencies that influence and are 
influenced by other agencies. Improvements are directed toward new 
practices not just transmitting new knowledge. Studies of evaluation as 
practiced by personnel in educational agencies are beginning to supply 
x>ntextual information important in planning for and implementing 
improvements in evaluation practice (c.f. Alkin et al. 1979? Kennedy, 
personal communication? Lyon, 1979? and Patton, 1978). 

Lyon's (1979) descriptive study of evaluation and school districts 
presents a number of specific findings related to the problem of multiple 
goals and the potential for conflict and ambiguity in facilitating 
evaluation use through technical assistance. 

First, evaluation units are relatively recent organizational 
arrangements. Fifty^one percent of the existing units sampled were 
organized or reorganized since 1970. Second, evaluation units typically 
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share authority with other units for data gathering. Third, local funds 
rather than federal or state monies dominate evaluation unit budgets* 
Fourth, there is little reported agreement either on what constitutes 
basic evaluation practice or on the priority of various evaluation 
activities. Fifth, larger districts tend to centralize evaluation units 
within administrative and support services divisions rather than include 
evaluation within instructional program units. Sixth, while non- 
evaluation staff perceive evaluation positions to be high on ambiguity 
and conflict, evaluation staff do not perceive such ambiguity and 
conflict to be an important aspect of their jobs. Heads of evaluation 
offices are more prone to see larger staffs, computer access and 
information about effective evaluation practices as ways of improving 
effectiveness. They do not view as particularly useful organizational 
changes,, improved communications or additional sstaff development programs. 

Lyon (1979) did not directly address the issues of evaluation use 
(i.e., what is it? how is it achieved?) in depth. Alkin et al. (1979) 
and Fatten (1978) have provided insights into these issues. Through 
their research, Alkin et al. (1979) and Patton (1978) have raised a 
"criterion problem" with respect to evaluation use. Given a general 
premise that evaluation is intended to increase rationality in program 
decision making, Alkin et al. (1979) contrast a "mainstream view" of 
evaluation use with an "alternative view" reflected in their research. 

The "mainstream view" was inferred from criticisms of many authors 
who have decried evaluations for not being used. Basic to the 
"mainstream view" is a belief that evaluation can a'xd should result in 
major program modifications or in make-or-break decisions on the programs 



assessed. The alternative view stresses a gradual and incremental impact 
of evaluations resulting from cumulative changes in decision makers' 
percepticms of programs. 

After analyzing case studies of five evaluation projects, Alkin et 
al. propose a conceptual framework for describing evaluation use. The 
framework suggests but does not establish "reality-based" criteria for 
using evaluation in local program improvement. Technical assistance 
designed to facilitate use of evaluation data for local program 
improvement should consider the implications of Alkin et al.'s (1979) 
descriptive framework of evaluation use as implying ultimate criteria for 
improving evaluation. 

Bach evaluation use requires communication of evaluation information 
to an appropriate user. The information may be used alone or in 
conbination with other influences to make decisions, substantiate 
pr^.vious decisions or actions or establish or alter attitudes related to 
a variety of programmatic issues (e.g., establishing a program, securing 
external funding, changing program components, securing ccsaaunity 
acceptance). 

Alkin et al. (1979) also identify a tentative analytic framework of 
29 properties affecting the process of evaluation use. These 29 
properties are organized within eight categories. A number of these 
properties are related to the influence of eKternally mandated evaluation 
activities (i.e., mandated bounds of an evaluation, use of a formal 
evaluation model, research and analysis considerations, the evaluators 
dealing with mandated evaluation tasks and influences of other 
governmental agencies). The case studies suggested that evaluation 
activities in "high use" districts tend not to be dominated by mandated 
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tasks, not to be based on for«.al evaluation models or rigorous design and 
analysis considerations. Supporting Patton (1978), Alkin et al. (1979) 
concluded that interpersonal factors involved in the evaluator's approach 
seemed to play a significant role in affecting use. 

The studies cited here are certainly not exhaustive of the area of 
field research on evaluation. They do seem to support David's (1978) 
contention that efforts to improve local use of Title I evaluation 
require more than changing the type or quality of information contained 
in the evaluations. Because of the need to examine organizational and 
interpersonal factors in evaluation use, an examination of the broader 
area of knowledge production and utilization is warranted. 
Theory and Research on Knowledge Production and Utilization 

The Research-Develop-D if fuse-Adopt (R-D-D-A) model has been, for a 
number of years, the dominant view of planned educational change. The 
R-D-D-A model, according to Cuba and Clark's (undatei) conception is 
initiated by research, followed by two stages of developnent (invention 
and design), which leads to two stages of diffusion (dissemination and 
demonstration), concluding in three stages of adoption (trial, 
installation and institutionalization). 

If one accepts the premise that evaluation of the TACs be viewed in 
terms of the R-O-D-A model as Oxford (1979) suggests, then the role and 
success of the TACs would be expressed in terms of their diffusion 
efforts (i.e., dissemination and demonstration) and aid given to states 
and districts as they adopt (i.e., conduct trial runs, install and 
institutionalize) the Title I Evaluation and Reporting System. But TACs 
have a broader mission as has been clearly established in policy. 
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Guba and Clark (1975)^ however, have questioned the adequacy of the 
R-D-'D-A model as a basis for developing "political support for 
educational R & D or to effect a significant improvement-^oriented change 
in educational practice** (p. 6). Representing a linear , unified systems 
view of R & the R-D-D-A model has a number of inadequacies* 

Application of the -.-D-D-A model is regarded as problematic because 
of a tendency toward: 

1* Establishing unachievable aspirations 

2. Ignoring the idiosyncratic (ideographic) goals of individuals 
anci individual agencies in the educational ocmimunity 

3. Changing policy and program directions persistently and 
frequently in an attempt to overcome failures in program 
achievement provoked by conditions 1 and 2 above. 

4. Overcentralizing and overcon trolling programs vdiich have been 
assessed as failures (Guba and Clark, 1975, p, 6). 

Guba and Clark call for reconceptualizing a broad approach to 
educational knowledge production and utilization (KPU). Such a 
conceptualization would account for the full spectrum of KPU functions 
(completeness), be responsive to individual, intraagency and interagency 
goals (balance) and reflect the reality of educational KPU (realistic). 
They describe a configurational via^ which is compared to the R-D-D-A 
model. The Guba and Clark (1975) analysis is particularly interesting 
because of the relationship between the issues regarding KPU processes in 
general and those involved in delivering technical assistance in 
evaluation. 
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The configurational view i& more flexible and "permissive it 
accounts for the full tange of KPU functions without assuming a linear 
flow of events. In theory the configurational view is superior to the 
R-D-D-A model. Flexibility and permissiveness, however, do not preclude 
the need for explicitness of innovations if improvement is to be 
acGomplished. An approach toward establishing explicitness without 
assuming elaborate predefinition and linearity is suggested by recent 
research on the impleanentation of innovations. 

Research on implementation provides clues for improving technical 
assistance that are grounded in "real events" rather than plans or 
intentions. In their review of re«3earch on curriculum and instruction 
implementation, Fullan and Porofret (1977) distinguish between planning , 
adoption and implementation . Planning is the process of specifying the 
intended use of an innovation? adoption is the decision to use the 
innovation; and implementation "refers to the actual use of an innovation 
or what an innovation consists of in practice" (Fullan and Pomfret, 1977, 
p. 336). The implementation process is generally ignored in the context 
of the R-D-D-A model. It is implicit, however, in the configurational 
view. 

In the case of Title I evaluation, each agency is involved in 
planning, adopting and implementing a set of Title I evaluation 
activities in accordance with one of the available models or an approved 
alternative. Idiosyncratic evaluation activities can and do exist within 
various agencies. To the extent that idiosyncracies are need-related, 
evaluation can be more influential in local program improvement. 

Since it is the Title I evaluation system as implemented by SEAs and 
LEAS that will produce the data for USOE's actual use (i.e., USOE-level 
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iraplementatloa) of the TIERS, local Title I evaluation needs to be 
oompatible uut not necessarily restricted by TIERS, This raises a second 
distinction made by Fullan and Porafret (1977) • They categorize studies 
of implementation as follc3wing one of two main orientations, the fidelity 
perspective and the mutuad adaptation perspective (see also Berman and 
Pauly, 1975, Berman and McLaughlin, 1976j and McLaughlin, 1976) • The 
emphasis in the fidelity perspective '•is to determine the degree of 
impl^entation of an innovation in terms of the extent to which actual 
use of the innovation corresponds to intended or planned use* {p. 340). 
The fidelity perspective is contrasted to the mutual adaptation 
perspective, which ^•is directed at analyzing the ccmiplexities of the 
change process vis-a-vis how innovations become developed/changed, etc. 
during the process of implementation" (p. 340). Under the former view^ 
sufficient definition of the innovation exists prior to diffusion while 
the latter view presumes the definition of the innovation emerges through 
the implementation process. 

These two perspectives represent alternative ways to conceptualize 
technical assistance roles as well as alternative perspectives on 
defining and describing the implementation of an innovation. 

A mutual adaptation perspective is called for when the sponsor has 
flexible impl^entation objectives. Local initiative is actually 
required. Planning and adoption choices need to be made. Therefore, 
both technical aissistance providers and receivers take on an active and 
diverse set of roles. 

External evaluations of the impl^entation of the USOE-sponsored 
efforts to improve Title I evaluatlcm may take either of the two 
perspectives. The fidelity perspective would entail describing what was 
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ijapLemented as compared to the Intentions of the sponsor. The mutual 
adaptation perspective would involve describing what was actually 
implemented. In either case, a descriptian is provided and a descriptive 

franewock is needed. 

One approach to describing implementation is provided by Hall and 
Loucks (1976). Pullan and Pomfret (1977) call it "the most sophisticated 
and explicit conceptualization of 'the fidelity* orientation to assessing 
degree of implementation" (p. 355). HaU and Loucks (1976) conceptualize 
impleaentation as progressing through a series of levels or stages. 
Implementation exists in terms of a continuum ranging from nonuse through 
renewal. The set of levels is as follows: 



0 


Nonuse 


I 


Orientation {initial information) 


II 


Preparation (to use) 


III 


Mechanical Use 


IVh. 


Routine Use 


IVB 


Ref ineroent 


V 


Integration 


VI 


Renewal 
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Operationalizing these levels in terms of Title T evaluation requires 
distinct criteria for USOE, SEAs and LEAs. One could envision 
descriptors of evaluation activities for various agencies. A suggested 
set of SEA and indicators of each level is presented in Table 1 and 
2, respectively. 
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Table I 

Levels o£ SEA Use of USOS Title I Evaluation and Repoctlng System 



Levels o£ 






Use* 




Indicators 


0 Nonuse 


• 


No knoi^ledge of TIERS 




• 


No involvement with TIERS 






Not seeking involvement 


1 Orientation 




Acquired or is acquiring information about TIERS 






Explored philosophical orientation of TIERS 




• 


Explored political or contractual application of TIERS 




• 


Explored decaands on SEA staff 




• 


Explored demands on tEAs 






Explored relationship o£ TIERS to In^piace Title I 






evaluation 


2 Preparation 




Planned first implementation oC TIERS 




• 


Provided orientation training to other SEA staff 




• 


Provided orientation to LEAs 




• 


Developed sampling plans 






Supplemented i«cia application torms 




• 


Defined range of LEA options 






Provided guidelines for LEA implementation of TIERS 




• 


Developed SEA analysis procedures 




• 


Developed SEA reporting system 


3 Mechanical 


• 


Effort focused on details of implementation 


Use 




Conmiuni cation to LEA concerning implementation 






decisions and options 




• 


Accepted feedback from LEA 




• 


Little time for reflection on overall use of TIERS 




• 


Changes or implementation decisions based more on 






user needs than client needs 




# 


Stepwise mastery of SEA tasks attempted 






Disjointed a d superficial use 



^Levels of use on Hall and LoucRs (1976) 
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Levels of 






Use* 




Indicators 


4 A Routine 


• 


Use of TIERS stabilized 


Use 




Pew if any changes 




• 


Little emphasis on improving use 






Little thought given to consequences of use 


4B Refine-- 




SEA varies TIERS components to increase ?ts use 


ment 




within imaediate sphere of influence 




• 


Variations occur in scheduling 




• 


Variations occur in implementation 




# 


Variations occur in analysis 






Variations occur in interpretation 




• 


Variations occur in reporting 






Variations are based upon reducing costs^ easing burden. 






USOE policy changes, technical information, consequences 






for LEAs increasing quality of information 


5 I nteg ration 


• 


Revisions to TIERS implementation based on other 






SEA evaluation efforts 






Revisions to other SEA evaluation efforts based on TIERS 






impl ementa t ion 






Emphasis in revision and refinements based upon 






achieving collective input 






Revisions are in accord with USOE policy 




• 


New evaluation questions begin to emerge 




m 


Competing or conflicting values surface 


6 Renewal 


• 


SEA re-evaluates quality of TIERS use 




• 


Major modifications of or alternatives to initial SEA 






TIERS application sought 






Emphasis on increased local use 






New technical developments examined 




• 


New goals for SEA Title I evaluation and for TIERS 



explored 
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Table 2 

Levels of hBPi Use of USOB Title I Evaluation and Reporting System 



Levels of 
Use* 


Indicators 


0 


Nonuse 


• No knowledge of TIERS 

# NO invoivsinent: wivn rxi^ra 
m Not seeking involvement 


1 


Orientation 


• Acquired or is acquiring information about TISRS 

• BKplored contractual application of TIERS 

• Explored demands on LEA 

• Explored relationship of TIERS to in-place Title I 
evaluation 

• Explored relationship of TIERS to in-place district 
testing program 


2 


Preparation 


• Selected an evaluation model 

• Provided orientation to appropriate Title I staff and 
other LEA staff and advisors 

• Assigned roles and responsibilities for TIERS 
implementation 

0 Developed a testing scneouie ana oraereo rescs 


3 


Mechanical 
Use 


• Effort focused on details of implementation 
m Execution of tasks monitored 

0 Data processed and filed; little feedback 

# Little reflection on use of TIERS data 


4A Routine Use 


• Use of TIERS stabilized 

• Pew, if any^ changes from year to year 

• Little emphasis on improving use 


4B 


Re £i nement 


• LEA fine-tunes TIERS evaluation to eliminate problems and 
improve timeliness 

m Changes are made in scheduling, implementation, reporting 
and interpretation 

• Changes are based on reducing costs losing burden, USOE 
policy changes, technical information and need for 
increased quality of information 

• LEA augments basic TIERS requirements to provide 
aditional information, such as diagnostic data* 
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Levels of 
Use* 


Indicators 


5 Integration 

6 Renewal 


• Revisions to Title I evaluation implementation based on 
other LEA evaluation efforts 

• Revisions to other LEA title I evaluation efforts related 
to TIERS impl mentation 

• Emphasis in revision and refinement based upon multiple 
district evaluation information needs 

• Local evaluaion questions begin to emerge 

• Evaluation data used in program improvement 

• LEA evaluates the quality of use of Title I evaluation 
data 

• Emphasis is placed on data use and on providing targeted 
information for decision makers 

• Options for improving Title I evaluation are explored and 
new approaches are planned and implemented 
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Assuming a progression through these levels vrould provide one means 
of forecasting SEA and hEh needa^ to prepare appropriate technical 
consultation strategies and materials^ One might argue that most states 
have gone through Preparation (Level II) and are presently at least at 
the stage of Mechanical Use. Other states that piloted the TIERS in 
advance of regulations may have progressed to Refinement (Level IVB) and 
may have begun £^anning for integration of Title I evaluation with other 
program operations or other evaluation activities* According to the Hall 
and Loucks continuum, use of evaluation for program improvement is an 
Indicator of the Integration level of use which is to be expected at a 
fairly advanced stage of implementation* 

Fullan and Porofret (1977) have questioned whether explicit definition 
of the levels of use should be attempted prior to implementation. This 
is a moot point in terms of Title I evaluation because to our knowledge 
such explicit definition and assessment has not heretofore been 
attempted* Nonetheless, an ad hoc assessment of current status in terms 
of the levels of use continuum may assist in planning technical 
assistance for SEAs or LEAs» The kinds of requests and the expectations 
a client agency has for TAG services should be related to the agency's 
orientation and current level of use* One problem needs to be addressed 
before such an assessment is attempted. As discussed previously, a 
variety of factors suggest that greater attention should be given to 
determining local evaluation needs early in the implementation process. 
Research on evaluation suggests a variety of perspectives exist among 
LEAs on what constitutes basic evaluation priori * nd that mandated 
evaluation activities tend to be secondary to other evaluation activities 
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in districts using evaluation infonaation* Local questions or issues may 
be raised prior to the Integration stage and should be dealt with 
whenever they arise « 

The oonf igurational view of the KPU implies there may be nonlinear ity 
or discontinuity across levels. A compliance orientation to 
implaoentation involves responding bureaucratically (i.e.r mechanically 
or routinely). Unless some internal motive emerges, the LEA or SEA may 
never move beyond the Routine Use level. 

Agencies with a current history of evaluation reform may view Title I 
evaluation as something that must fit in with ongoing attempts to improve 
evaluation. These agencies may be motivated more toward refinement goals 
(e.g., using evaluation for program improvement) and compatibility with 
internal efforts than toward meeting external requirements. They simply 
may not perceive themselves as progressing through a series of levels 
initiated by the introduction of the TIERS. Under these circumstances, 
application of the linear impl^entation model may suggest to the client 
that goals are not appreciated or understood by the provider or sponsor 
of technical assistance. 

A Conceptual Model for Analyzing SBA implementation 

The following conceptual model is intended to illustrate this problem 
in two-<*dimensional form. 

Quadrant I represents the client group with ili-»formed Title I 
implementation objectives. There Is little internally or externally 
based motive to change current Title I evaluation practice. TAC 
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High Compliance 



Low Program Improvement 




High Program Improvement 



Low Compliance 



Figure 1: A Two-Dimensional Model of Title I Evaluation IraplainGntation 
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activities directed toward such client groups would involve outreach and 
awareness level coram uni cat ion- According to the Hall and Loucks (1976) 
continuum described earlier, these client groups wo Id be in a state of 
Nonuse • 

Quadrant II represents the client group with impl mentation 
objectives established prior to the introduction of the mandated 
evaluation. These client groups must consider adapting their current 
evaluation strategies to the mandate introduced. TAG services to these 
client groups would Include outreach and awareness plus needs sensing, 
problem solving assistance including that unrelated to implementation of 
the mandate. These states or districts must carefully avoid increasing 
the perceived cGiQplexity of their change efforts. Such complexity may 
detract from local implementation (Pullan and Pomfret, 1977) and utility. 

Under the configural view and the mutual adaptability perspective, 
TACs need to be particularly sensitive to conflicts between the mandated 
system and local evaluation goals. TACs could serve to facilitate 
negotiations betv^een the external developer and the client group. 

Placing a Quadrant II client group in the levels of use continuum is 
probl«natic. with respect to TIERS they would probably be at the Nonuse 
stage. With respect to their internally defined improvement efforts they 
oould be at any level. The more advanced they au:e with respect to the 
Internal effort, the more likely they could Incorporate a new Innovation, 
assuming it does not conflict in Important ways with their current 
system. This hypothesis is baaed on conclusions of Rogers and Shoemaker 
(1971) that relative simplicity of an Innovation and Its compatibility 
with current practice contribute to effective implementation. 
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Quadrant III represents the client group whose Title I evaluation 
implementation is well established and bounded by the mandated evaluation 
specified in the law and regulations. TAC activities would emphasize 
orientation, model selection, model implementation and reporting. These 
client groups would be at the Orientation, Preparation, Mechanical Use or 
Routine Use levels. 

Quadrant IV contains client groups whose implementation is not 
bounded by the mandated system but includes it as an integral element. 
Implementation o£ the TIERS is an incc^nplete expression o£ client needs, 
intentions and goals. 

TAC services to Quadrant IV clients include all of those appropriate 
for Quadrant III clients, as well as needs sensing, problem 
identification, goal clarification and technical support during 
implementation. Client groups in Quadrant IV may range from the 
Orientation level to the Renewal level of use. Their overall goal is 
guided toward Renewal. 

If we were to follow either the R-D-D-A Model of early Cuba and Clark 
(undated) or the linear interpretation of levels of use hypothesized by 
Hall and Loucks (1976), forecasting based on movement within or between 
quadrants would be problematic. Clients beginning in Quadrant II are one 
important perturbation. Assuming a linear progression beginning with 
Nonuse and moving to a predefined level (e.g., Routine Use) would not 
explain movement of states beginning in this quadrant. 

Conceptualizing implementation intent as involving two independent 
dimensions reflects the essentially dynamic, nonlinear view of change 
implicit in the conf igurational view. It suggests greater role 



complexity 2or technical assistance • Technical assistance must use a 
broader range of roles than provided by the linear R-D-D-A Model • 

An integration of the Hall and Loucks (1976) continuum and the 
two-dimensional model of orientation toward Title I evaluation 
implementation is presented in Figure 2. This model provides a 
conceptual scheme that can be translated into operational terms within 
each receiver agency, 

In^l^S £ia^£®-2_Her e 

Whether clients are SBAs or LEAs, placing them on the two-dimensional 
grid raises several questions: 

1» Is there differentiation in TAG activity according to the 

quadrant in which the client is perceived to fall? 
2* Is the quadrant placement of clients stable or shifting over 

time? If it is shifting ^ what seems to stimulate movement, and 
how does movement occur? 
3* Do change constructs represented have utility to TAG or to 

clients in planning? 
In answer to the first question, TAG services can be described in 
terms of function, mode and scheduling. Three functions that account for 
most of TAG'S direct service to clients are conveying information, 
training clients in specific technical skill areas, such as how to 
implement an evaluation model, and helping clients in planning and 
problai^ solving related bo evaluation. Further specifications of these 
service functions are provided in Table 3» 
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High Compliancd 



Low Program Improvement 



NoRuse 




Renewal 



High Program improvement 



Low Compliance 



Figure 2: A Three-Dimensional Model of Title I Evaluation Irapleraentation 



33 



35 



ERIC 



Table 3 



Examples Of Content Of Technical Consultation By TAC 



TAC 

Consultation 
Function 

Conveying 



SEA & LEA Evaluation Activities 



rianning 



Implementing 



Analysing 



Interpreting 



Reporting 



Providing raate- 
reials (e.g* # 
test charts) to 
LEAS for plan- 
ning an evalu- 
tion» 



Providing imple- 
mentation checks- 
lists for each 
evaluation model* 



Making available 
to project direc- 
tors directions 
for the use of 
programmable 
calculators* 



Corresponding 
with an LEA to 
assist in inter- 
preting an 
unanticipated 
or anomalous 
result. 



Providing a paper 
with suggestions 
for effective 
reporting tech- 
niques. 



Helping Consulting with 

an LEA to help 
integrate Title I 
^ testing into the 

district testing 
program* 



Assisting an LEA 
develop selection 
criteria, includ- 
ing composite 
scores. 



Consulting with 
the SEA to ana- 
lyze errors on 
LEA report forms 
and on Title I 
computer print- 
out* 



Reviewing evalu- 
ation results 
with an LEA to 
recommend pror- 
gram modifica- 
tions. 



Assisting an LEA 
develop an evalu- 
ation report for 
a JDRP proposal. 



Training 



Providing a work- Training SEA 



shop to PAC 
representatives 
so they can 
participate in 
evaluation 
planning. 



Title I staff on 
nuances of model 
implementation. 



Teaching LEA 
representatives 
to make NCE and 
out-of-level 
score 

conversions. 



Conducting a 
workshop for LEA 
staff on the use 
of item data for 
instructional 
planning. 



Conducting a 
simulation 
activity on data 
reporting. 
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The technical services are carried out by TAC via in-pecson or 
telephone consultations, workshops and media presentations — both print 
and audio- visual. These modes are often used in combination. TAC 
services to various clients also vary in terms of the frequency, duration 
and re9ularity of contact. Annual training workshops, ad hoc helping 
consultations by telephone, and quarterly newsletters conveying 
information are illustrations of TAC activities that vary in scheduling. 

Row might services described in terns of these three categories 
differ for clients characterized in terras of their quadrant placement? 
The degree to which a client's evaluation implementation goals are 
compliance-oriented should correlate with the amount of technical 
assistance sought for conveying and training activities; the 
self- improvement dimension to the amount of technical assistance for 
helping activities. These activities are correlated with delivery mode. 
Conveying is achieved efficiently with media, often used in a workshop 
setting; training likewise is effectively carried out through workshops 
with reliance placed on simulation materials. Helping activities, on the 
other hand, involve an interactive process combining responsive and 
proactive elements. 

The predicted relationships between quadrant placement and dominant 
characteristics of technical assistance are summarized In Table 3, and 
suggest differences among quadrants In terms of activity, mode and 
scheduling. Data submitted to the U.S. Office of Education for the first 
three years of the TAC contracts tend to support some of the hypothesized 
relationships. 

Insert Table 4 Here 
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Table 4 



Predicted Relationship Between Model 
Placement and Characteristics oC Technical j\ssistdnce 





Major 


Dominant Characteristics o£ TA 


Quadrant 


ProbIeiQ8/Ne<H3s 


Activity 


Mode 


Scheduling 


T 


tonoed 






T n t* i^tfm t A n 
X 11^ r f;%IU^n V 




implementation 






Short duration 




objectives 






Irregularly scheduled 




Low motivation 










vo w n a n^e e Va x*^ 










uation practices 








IT 


Need to adapt 


Helping 


Consultation 


Frequent 




evaluation 






Variable duration 




strategies to 






Regular and ad hoc 




mandate 






schedule 




Need to avoid 










complexity 








III 


Need to conform 


Conveying/ 


Workshop 


Moderate frequency 




to mandate 


Training 




Longer duraticm 










Regularly scheduled 


IV 


Need to meet 


Conveying/ 


Workshop & 


Frequent 




additional 


Training/ 


Consultation 


Variable duration 




evaluation 


Helping 




Regular and ad hoc 




goals 






schedule 



Need to utilize 
evaluation data 
for program 
improvement 
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The data base contains information about mode and frequency, not 
about function, duration of contact or regularity of scheduling. Other 
limitations of the data base include incompleteness, inconsistency over 
tiae of the recording system and method of aggre.ation which for our 
purposes probably obscures the pattern of predicted relationships. Since 
the data are judged not to be of sufficiently high quality to subject to 
rigorous analysis, the approach taken was to seek out e^^amples and 
counterexamples of th«» relationships predicted, in or ler to make a 
preliminary judgment of the potential for pursuing the hypotheses stated 
more formally. 



Insert Figure 3 Here 
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Inspection o£ Figures 3a, 3b and 3c reveals that Quadrant III SEAs 
used workshops more than other modes of technical assistance. The 
differences between Quadrants II and IV are less pronounced, though there 
is slightly greater relianr* i^Jon consultation in Quadrant II, as 
predicted. 

The second question concerned the stable or shifting placement of 
clients over time. Clearly, with respect to TIERS, all clients began in 
Quadrants I or II in 1976. We suggest that movement can be meaningfully 
described in terms of the levels of use continuum and that for most 
states or districts beginning in Quadrant I, movement has been linear. 
States or districts that began in Quadrant 11 will depart from linear 
impl«Bentation across levels. To date, evidence regarding levels of use 
is indirect and based on informal observation. 

Let us now turn to the final question, do the constructs presented in 
this paper have utility to TAG or its clients in planning? 
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Figure 3a 




Figure 3c 
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Figure 3: Mode of Field Contact for Five SEAs, 1976-79 
(3a, Quadrant II; 3b, Quadrant III; 3c Quadrant IV) 
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Historically IT planning occurs when TACs plan with SEAs annually as 
letters of agreement are negotiated* Planning has frequently been 
event-oriented; £or example, an annual state or regional Title I 
conference is scheduled, at which TAG assistance is requested. Modes and 
scheduling of technical assistance to be provided are often estimated on 
the basis of past experience combined with current projections. The 
content focus of the technical assistance, is not necessarily fully 
explicated ror is it related to the negotiated agreements specifying mode 
and amount of technical assistance. Content selection may be made ttom a 
smorgasbord offering of a variety of topics related to TIERS, testing, 
program evaluation and ancillary services. 

In the absence of specific client goals for improving Title T 
evaluation efforts or a common framework for analyzing status of 
impl mentation, such planning is often based on best guesses for what 
services are needed. In this situation, the providers of technical 
assistance might encourage receivers to look at their position on the two 
dijnensional model of implementation and the level of use framework, 
thereby assessing their status and clarifying their evaluation goals. 
Such self-appraisal can lead to the planning of efficient, timely and 
client-centered technical assistance activities. 

These models can be used as heuristic guides or more formally as a 
basis tot establishing systematic means of data collection. On the 
hueristic level, simple presentation of either model has been observed to 
heighten clients' awareness of their implicit implementation goals and 
strategy and has provided a structure that complements the presentation 
of typical listings of TAC workshops and consultations. In one instance 
an SEA client's concerns have been observed to shift in emphasis as a 
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resuLt of pointing out that previous services Eocaased primarily on 
compliance over program improvement {e*g», the SBH perceived itself to be 
in Quadrant III and to have moved in a vertical plane with little 
horizontal movement). 

Several informal presentations of the levels of use oontinuum have 
elicited such client reactions as: **This is where I am and this is where 
I want to be.* 

More systematic application of the levels of use framework involves 
interview or questionnaire development to allow for assessment of 
status* Through self-* assessment an SBA may determine its location alwig 
the implementation continuum. Interim evaluation system objectives can 
be established within an overall implementation framework. Retracking to 
plug gaps in the overall scheme can be considered. 

Roles and responsibilities of SEAs, LBAs and TACs can be established 
within such a framework. Products documenting and solidifying level 
attainment can be prepared in a timely fashion. Receivers and providers 
can better forecast critical issues associated with progressing across 
levels. 

While the model is presented in linear terms it is recognised that 
certain events or demands may break down the apparent linearity. When 
this happens , however ^ the SEA (or LEA) can evaluate the implications of 
such departures in terms of their readiness or the readiness of their 
LBAs. 

Instrumentation could also be used to document (across time) gradual 
improvements in Title I evaluation resulting from successful integration 
of mandated and self- initiated evaluation activities. 
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